7/9

# 4 Configuring Oracle Exadata Database Machine for Use with Oracle Big Data Appliance

This chapter provides information about optimizing communications between Oracle Exadata Database Machine and Oracle Big Data Appliance. It contains the following sections:

- About Optimizing Communications

- Prerequisites

- Enabling SDP on Exadata Database Nodes

- Configuring a JDBC Client for SDP

- Creating an SDP Listener on the InfiniBand Network

## 4.1 About Optimizing Communications

Sockets Direct Protocol (SDP) is a standard communication protocol for clustered server environments, providing an interface between the network interface card and the application. By using SDP, applications place most of the messaging burden upon the network interface card, which frees the CPU for other tasks. As a result, SDP decreases network latency and CPU utilization, and thereby improves performance.

This chapter describe how you can configure Oracle Exadata Database Machine to use SDP over InfiniBand to communicate with Oracle Big Data Appliance.

## 4.2 Prerequisites

Oracle Big Data Appliance and Oracle Exadata Database Machine racks must be cabled together using InfiniBand cables. The IP addresses must be unique across all racks and use the same subnet for the InfiniBand network.

See Also:

- Oracle Big Data Appliance Owner's Guide about multirack cabling

- Oracle Big Data Appliance Configuration Worksheets about IP addresses and subnets

## 4.3 Enabling SDP on Exadata Database Nodes

The following procedure describes how to enable SDP on database nodes in an Oracle Exadata Database Machine running Oracle Linux.

To enable SDP on Oracle Exadata Database Machine:

1. Open /etc/infiniband/openib.conf file in a text editor, and add the following line:

       set: SDP_LOAD=yes

2. Save these changes and close the file.

3. To enable both SDP and TCP, open /etc/ofed/libsdp.conf in a text editor, and add the `use both` rule:

       use both server * :
       use both client * :

4. Save these changes and close the file.

5. Open `/etc/modprobe.conf` file in a text editor, and add this setting:

       options ib_sdp sdp_zcopy_thresh=0 recv_poll=0

6. Save these changes and close the file.

7. Replicate these changes across all servers in the Oracle Exadata Database Machine rack.

8. Restart all database nodes for the changes to take effect.

9. If you have multiple Oracle Exadata Database Machine racks, then repeat these steps on all of them.

## 4.4 Configuring a JDBC Client for SDP

The following procedure explains how to configure a JDBC client to use SDP.

To enable SDP support for JDBC:

1. Configure the database to support InfiniBand, as described in the Oracle Database Net Services Administrator's Guide. Ensure that you set the protocol to SDP.

2. Set the `LD_PRELOAD` environment variable to libsdp.so before starting the Java virtual machine. This example uses the Bash shell:

       export LD_PRELOAD="libsdp.so"

The following steps are an alternative to setting `LD_PRELOAD`:

1. In the JDBC URL, replace `TCP` protocol with `SDP` protocol. For example:

       jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=xxx.x.x.x)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice)))

2. Open DOMAIN_HOME/bin/startWebLogic.sh in a text editor and make the following change:

   1. Locate this line in the file:

      `. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*`

   2. Add this property immediately after the previous line:

          JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true  -Doracle.net.SDP=true"

   3. Save and close the file.

## 4.5 Creating an SDP Listener on the InfiniBand Network

To add a listener for the Oracle Big Data Appliance connections coming in on the InfiniBand network, first add a network resource for the InfiniBand network with virtual IP addresses.

Note:

This example lists two nodes for an Oracle Exadata Database Machine quarter rack. If you have an Oracle Exadata Database Machine half or full rack, you must repeat node-specific lines for each node in the cluster.

1. Edit `/etc/hosts` on each node in the Exadata rack to add the virtual IP addresses for the InfiniBand network. Make sure that these IP addresses are not in use. For example:

       # Added for Listener over IB
       192.168.10.21 dm01db01-ibvip.example.com dm01db01-ibvip
       192.168.10.22 dm01db02-ibvip.example.com dm01db02-ibvip

2. As the `root` user, create a network resource on one database node for the InfiniBand network. For example:

       # /u01/app/grid/product/11.2.0.2/bin/srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/bondib0

3. Verify that the network was added correctly with one of the following commands:

       # /u01/app/grid/product/11.2.0.2/bin/crsctl stat res -t | grep net
       ora.net1.network
       ora.net2.network -- Output indicating new Network resource

   or

       # /u01/app/grid/product/11.2.0.2/bin/srvctl config network -k 2
       Network exists: 2/192.168.10.0/255.255.255.0/bondib0, type static -- Output indicating Network resource on the 192.168.10.0 subnet

4. Add the virtual IP addresses on the network created in Step 2, for each node in the cluster:

       # srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.255.0/bondib0 -k 2
       # srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.255.0/bondib0 -k 2

5. As the `oracle` user, who owns Grid Infrastructure Home, add a listener for the virtual IP addresses created in Step 4.

       # srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522

6. For each database that will accept connections from the middle tier, modify the `listener_networks init` parameter to allow load balancing and failover across multiple networks (Ethernet and InfiniBand). You can either enter the full `TNSNAMES` syntax in the initialization parameter or create entries in `tnsnames.ora` in the `$ORACLE_HOME/network/admin` directory. The `TNSNAMES.ORA` entries must exist in `GRID_HOME`. The following example first updates `tnsnames.ora`.

   Complete this step on each node in the cluster with the correct IP addresses for that node. `LISTENER_IBREMOTE` should list all other nodes that are in the cluster. `DBM_IB` should list all nodes in the cluster.

   Note:

   The `TNSNAMES` entry is only read by the database instance on startup, if you modify the entry that is referred to by any `init.ora` parameter (`LISTENER_NETWORKS`), you must restart the instance or issue an `ALTER SYSTEM SET LISTENER_NETWORKS` command for the modifications to take affect by the instance.

       DBM =
       (DESCRIPTION =
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521))
       (CONNECT_DATA =
       (SERVER = DEDICATED)
       (SERVICE_NAME = dbm)
       ))

       DBM_IB =
       (DESCRIPTION =
       (LOAD_BALANCE=on)
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522))
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522))
       (CONNECT_DATA =
       (SERVER = DEDICATED)
       (SERVICE_NAME = dbm)
       ))

       LISTENER_IBREMOTE =
       (DESCRIPTION =
       (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip.mycompany.com)(PORT = 1522))
       ))

       LISTENER_IBLOCAL =
       (DESCRIPTION =
       (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522))
       (ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522))
       ))

       LISTENER_IPLOCAL =
       (DESCRIPTION =
       (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm0101-vip.mycompany.com)(PORT = 1521))
       ))

       LISTENER_IPREMOTE =
       (DESCRIPTION =
       (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan.mycompany.com)(PORT = 1521))
       ))

7. Connect to the database instance as `sysdba`.

8. Modify the `listener_networks init` parameter:

       SQL> alter system set listener_networks=
            '((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)
               (REMOTE_LISTENER=LISTENER_IBREMOTE))',
            '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)
               (REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;

9. Restart `LISTENER_IB` to implement the modification in Step 7:

       # srvctl stop listener -l LISTENER_IB
       # srvctl start listener -l LISTENER_IB